Systems and methods for quantifying concrete surface roughness

ABSTRACT

The degree of concrete surface roughness contributes to the bond strength between two concrete surfaces for either new construction or repair and retrofitting of concrete structures. Provided are novel systems and methods with industrial application to quantify concrete surface roughness from images which may be obtained from basic cameras or smartphones. A digital image processing system and method with a new index for concrete surface roughness based on the aggregate area-to-total surface area is provided. A machine learning method applying a combination of advanced techniques, including data augmentation and transfer learning, is utilized to categorize images based on the classification given during the learning process. Both methods compared favorably to a well-established method of 3D laser scanning.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a divisional application of U.S. applicationSer. No. 17/373,057, filed Jul. 12, 2021, the disclosure of which ishereby incorporated by reference in its entirety, including all figures,tables, and drawings.

GOVERNMENT SUPPORT

This invention was made with government support under 69A3551747121awarded by US Department of Transportation. The government has certainrights in the invention.

BACKGROUND

The degree of concrete surface roughness contributes to the bondstrength between two concrete surfaces which are cast at different agesfor either new construction or repair and retrofitting of concretestructures. Many methods are presented in the literature to estimate thedegree of concrete surface roughness either qualitatively orquantitatively; however, most of these methods present drawbacksincluding being a contact method, being expensive, needing excessivehuman processing, and not being suitable for assessing old structuresbased on their inspection records.

BRIEF SUMMARY

Embodiments of the subject invention provide novel and advantageoussystems and methods for quantification of concrete surface roughnessfrom images that may be obtained from basic cameras or smartphones. Adigital image processing method with a new index for concrete surfaceroughness based on the aggregate area-to-total surface area (AR) isintroduced. A machine learning method applying a combination of advancedtechniques, such as data augmentation and transfer learning, is utilizedto categorize images based on the classification given during thelearning process. These novel methods were related to a well-establishedmethod of 3D laser scanning from sandblasted small concrete surfaces.Additionally, new images from the web of a sandblasted large-scaleT-beam have been used to examine and validate both methods.

In an embodiment, a system for determining a measurement of surfaceroughness of a concrete sample can comprise a processor and a(non-transitory) machine-readable medium in operable communication withthe processor and having instructions stored thereon that, when executedby the processor, perform the following steps: receiving an image of theconcrete sample; defining a positive integer n and a positive integerindex i ranging from 1 to n; defining a set of n roughness classes (Ci);defining for each Ci an associated average roughness value (Ci_(av));generating for each Ci, a probability (P_(i)) of matching the image withthat respective Ci; and determining a weighted average roughness value(R_(a)) for the image from the sum of each P_(i) multiplied by therespective Ci_(av) to obtain the measurement of surface roughness of theconcrete sample. The determining of the R_(a) for the image can compriseusing the following equation:

$R_{a} = {\sum\limits_{i = 1}^{n}{\left( {\left( P_{i} \right) \cdot \left( {Ci_{av}} \right)} \right).}}$The generating of the P_(i) can comprise using a convolutional neuralnetwork. The convolutional neural network may be trained using atransfer learning technique. The convolutional neural network may betrained using a data augmentation technique. The data augmentationtechnique may be applied in an offline manner to increase the samplesize of training data and can comprise one or more of random left andright rotation of images, change in brightness, blur with a uniformfilter, horizontal and vertical flipping, or resizing of images. Theconvolutional neural network may be trained using both a transferlearning technique and a data augmentation technique. The convolutionalneural network may be trained using labeled surface roughness values oftraining images measured by an independent roughness measurement method.The independent roughness measurement method can comprise a non-contactmeasurement. The non-contact measurement can comprise 3D laser scanningand/or digital image analysis.

Embodiments may provide systems and methods for obtaining a measurementof surface roughness of a concrete sample. The systems and methods cancomprise: receiving, by a processor, a first training image set;augmenting, by the processor, the first training image set, to create anaugmented training image set which can comprise training images;determining, by the processor, a surface roughness measurement for eachtraining image; classifying, by the processor, each training image intoa training classification group based on the surface roughnessmeasurement determined for that respective training image; determining,by the processor, an average surface roughness measurement for eachtraining classification group; training, by the processor, aconvolutional neural network to classify new images into a roughnessclassification group, to produce a trained convolutional neural network(TCNN); receiving, by the processor, a new image of the concrete sample,the new image not being present in the augmented training image set; anddetermining, by the processor, using the TCNN, a value of averagesurface roughness (R_(a)) for the new image of the concrete sample, toobtain the measurement of surface roughness of the concrete sample.

The augmenting of the first training image set can comprise one or moreof random left and right rotation of images, change in brightness, blurwith a uniform filter, horizontal and vertical flipping, or resizing ofimages. The determining of a surface roughness measurement for eachtraining image can comprise at least one of loading values from existingmeasurements or calculating one or more values from digital imageprocessing of training images. The existing measurements can comprisenon-contact measurements associated with respective training images. Thedigital image processing of training images can comprise calculation ofa ratio of aggregate area-to-total surface area (AR). The training ofthe convolutional neural network can comprise transfer learning from adataset larger than the augmented training image set and training ontraining images from the augmented training image set. The determiningof R_(a) for the new image of the concrete sample may be completed on acomputing device (e.g., a mobile device, phone, tablet, or laptopcomputer) at the same physical location as the concrete sample. The newimage of the concrete sample may be obtained from a camera in operablecommunication with the computing device at the same physical location asthe concrete sample.

A camera in operable communication with the mobile device may include anintegrated camera of a mobile phone, tablet, or laptop; or a separatecamera connected by wired or wireless connection (e.g., Bluetooth, nearfield communication, USB, serial cable, or other connection methodsknown in the art) or by transfer of media (e.g., removing an SD cardfrom a phone and placing the card in a reader on or attached to themobile device.)

The same physical location as the concrete sample may include a jobsite, construction site, roadway, repair location, building site, orinvestigation site. The same physical location as the concrete samplemay include locations where an action is to be taken or a decision madewith respect to the concrete sample (e.g., determining a need forfurther surface preparation, enhancing roughness degree, adding amechanical connector, or applying a bonding agent) within a period oftime (e.g., within minutes, in the same hour, day, or trip) frompreparing the concrete sample or obtaining the image of the concretesample. The same physical location as the concrete sample may includelocations within a specified distance, e.g., within 1 kilometer,alternatively within 0.1, 0.25, 0.5, 2, 5, or 10 kilometers, includingincrements, combinations, and ranges of any of the foregoing.

Embodiments provide systems and methods for determining a measurement ofsurface roughness of a concrete sample. The systems and methods cancomprise a processor and a (non-transitory) machine-readable medium inoperable communication with the processor and having instructions storedthereon that, when executed by the processor, perform the followingsteps: receiving an image of the concrete sample; defining a positiveinteger n and a positive integer index i ranging from 1 to n; defining aset of n roughness classes (Ci), defining for each Ci an associatedaverage roughness value (Ci_(av)); generating for each Ci, a probability(P_(i)) of matching the image with that respective Ci; and determining aweighted average roughness value (R_(a)) for the image from the sum ofeach P_(i) multiplied by the respective Ci_(av) to obtain themeasurement of surface roughness of the concrete sample. The determiningof the R_(a) for the image can comprise using the following equation:

$R_{a} = {\sum\limits_{i = 1}^{n}{\left( {\left( P_{i} \right) \cdot \left( {Ci_{av}} \right)} \right).}}$The generating of the P_(i) can comprise using a convolutional neuralnetwork. The convolutional neural network may be trained using atransfer learning technique. The convolutional neural network may betrained using a data augmentation technique. The data augmentationtechnique may be applied in an offline manner to increase the samplesize of training data and can comprise one or more of random left andright rotation of images, change in brightness, blur with a uniformfilter, horizontal and vertical flipping, or resizing of images. Theconvolutional neural network may be trained using labeled surfaceroughness values of training images measured by an independentnon-contact roughness measurement method which can comprise either 3Dlaser scanning and/or digital image analysis.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is an image of a concrete surface roughened by sandblasting.

FIG. 1B is a two-dimensional representation of a scanned surface withpeaks and valleys and a best fit plane produced from a 3D laser scanningmethod.

FIG. 2A is a full color RGB image of a concrete sample ready for imageprocessing according to an embodiment of the subject invention.

FIG. 2B is an RGB image after applying a contrast enhancement filter forimage processing according to an embodiment of the subject invention.

FIG. 2C is a segmentation of the RGB image after applying contrastenhancement filter for image processing according to an embodiment ofthe subject invention.

FIG. 2D is a segmented gray-scale image for image processing accordingto an embodiment of the subject invention.

FIG. 2E is a segmented image in the process of threshold selection forimage processing according to an embodiment of the subject invention.

FIG. 2F is a black and white image showing separation of aggregate forimage processing according to an embodiment of the subject invention.

FIG. 3A is an image of concrete surface in office light used inaccordance with embodiments of the subject invention.

FIG. 3B is an image of concrete surface in a dark room with flash usedin accordance with embodiments of the subject invention.

FIG. 3C is an image of concrete surface from a scanner used inaccordance with embodiments of the subject invention.

FIG. 3D is an image of concrete surface from an angle used in accordancewith embodiments of the subject invention.

FIG. 4A is a sample of an original image of a concrete surface used inaccordance with embodiments of the subject invention.

FIG. 4B is a sample of an augmented flipped original image of a concretesurface used in accordance with embodiments of the subject invention.

FIG. 4C is a sample of an augmented flipped image of a concrete surfaceused in accordance with embodiments of the subject invention.

FIG. 4D is a sample of an augmented blurry image of a concrete surfaceused in accordance with embodiments of the subject invention.

FIG. 5 is a block diagram of the ResNet model used in accordance withembodiments of the subject invention.

FIG. 6A shows sample image 1 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6B shows sample image 2 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6C shows sample image 3 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6D shows sample image 4 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6E shows sample image 5 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6F shows sample image 6 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6G shows sample image 7 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6H shows sample image 8 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 6I shows sample image 9 of 9 used to test a digital imageprocessing method and a machine learning method according to embodimentsof the subject invention.

FIG. 7 is a chart showing a correlation between the ratio of theaggregate area-to-total area (AR) to concrete surface roughness (R_(a)).

FIG. 8 is a chart showing the accuracy of augmented and non-augmenteddeep learning models at each epoch.

FIG. 9 is a chart showing the loss values of augmented and non-augmenteddeep learning models at each epoch.

FIG. 10 is an illustration of a large scale (H=419 mm; L=4,724 mm;X=203.2 mm; Y=203.2 mm) T-beam with roughened surface with six samplesmarked for analysis.

FIG. 11A shows large-scale sample image 1 of 6 extracted from the beamrepresented in FIG. 10 and used to test a digital image processingmethod and a machine learning method according to embodiments of thesubject invention.

FIG. 11B shows large-scale sample image 2 of 6 extracted from the beamrepresented in FIG. 10 and used to test a digital image processingmethod and a machine learning method according to embodiments of thesubject invention.

FIG. 11C shows large-scale sample image 3 of 6 extracted from the beamrepresented in FIG. 10 and used to test a digital image processingmethod and a machine learning method according to embodiments of thesubject invention.

FIG. 11D shows large-scale sample image 4 of 6 extracted from the beamrepresented in FIG. 10 and used to test a digital image processingmethod and a machine learning method according to embodiments of thesubject invention.

FIG. 11E shows large-scale sample image 5 of 6 extracted from the beamrepresented in FIG. 10 and used to test a digital image processingmethod and a machine learning method according to embodiments of thesubject invention.

FIG. 11F shows large-scale sample image 6 of 6 extracted from the beamrepresented in FIG. 10 and used to test a digital image processingmethod and a machine learning method according to embodiments of thesubject invention.

FIG. 12 is a chart showing a comparison between a 3D laser scanningmethod, an image processing method according to an embodiment of thesubject invention, and a machine learning method according to anembodiment of the subject invention.

FIG. 13 shows a value of concrete surface roughness index AR for each ofthree different concrete surfaces measured according to an embodiment ofthe subject invention.

FIG. 14 shows digital image processing results, AR, used to classify andlabel images from FIGS. 6A through 6I into three categories for themachine learning training phase according to an embodiment of thesubject invention.

FIG. 15 shows a confusion matrix for deep learning model with transferlearning and data augmentation (“aug.” model).

FIG. 16 shows a confusion matrix for deep learning model with transferlearning and without augmentation (“non-aug.” model).

DETAILED DESCRIPTION

Embodiments of the subject invention provide novel and robustnon-contact methods to evaluate concrete surface roughness from images(e.g., smart phone photos) using a machine learning approach. In thisapproach deep learning (e.g., deep neural networks) has been utilized toextract deep and meaningful features directly from raw images withoutany manual feature extraction or human intervention.

Embodiments provide a machine learning method, which may be defined as aclassification technique that predicts the class of each image based onits deep visual features. Classification is a supervised machinelearning method that requires data with pre-assigned labels. The modelcan learn from the existing data samples to predict the labels of futureimages. Available images may be divided into training sets and testingsets. A training set includes the data that are used for training themachine learning model while a testing set includes the data forvalidating the model. In some embodiments, approximately 80% of imagesmay be used for training and 20% of images may be utilized for testing.The success of machine learning and deep learning techniques in the areaof image and visual data processing heavily depends on the availabilityof large scale annotated datasets to learn the existing patterns in thedata. However, collecting large scale image datasets with labels may betime consuming, tedious, and expensive especially in applications wheresample size may be small and access to samples may be limited (e.g., incivil engineering an infrastructure applications generally, or inexamination, quantification, analysis or repair of concrete structuresspecifically.) Embodiments of the subject invention may advantageouslyapply a powerful technique called data augmentation to generatesynthetic training images from the existing data. This method helps thedeep learning model to be generalized to new conditions and environments(e.g., those that are never experienced beforehand.) If the model onlytrains on the current image dataset, it may be difficult or almostimpossible to predict the class of new images created under differentconditions (e.g., images which are taken in a different lightingcondition, or with a different angle.) However, using data augmentation,it is possible to generate a variety of samples by changing differentcharacteristics of the images. Embodiments provide one or moreaugmentation operations including random rotation, blur, brightness,horizontal flipping, vertical flipping, and resizing the images. Theseoperations are applied only on the training data to increase the samplesize.

Casting concrete at different ages for new construction and forrepairing or retrofitting concrete structures requires a sufficient bondbetween concrete casts. The bond strength between different casts isattributed to surface roughness. Surface roughness can be achieved inmany ways, such as water jetting or sandblasting. In order to evaluatethe degree of surface roughness, qualitative and quantitative methodsare introduced by many researchers; however, several drawbacks areassociated with these methods, including cost, availability, humanerrors, and inability to assess old structures from prior inspectionrecords. Embodiments of the subject invention provide novel industrialimplementation methods to quantitatively estimate the concrete surfaceroughness from images with sufficient resolution. In a first applicationmethod, a digital image processing method is proposed to distinguish thecoarse aggregate from cement paste, and a new index is presented as afunction of aggregate area proportional to the total surface area. In asecond application method, data augmentation and transfer learningtechniques in computer vision and machine learning are utilized toclassify new images based on predefined images during the learningprocess. Both application methods are compared to a well-establishedmethod of 3D laser scanning from sandblasted concrete surfaces. A brandnew set of images of sandblasted surfaces have been used to test andvalidate these novel methods. The results show that both tested methodssuccessfully estimate the concrete surface roughness with an accuracy ofmore than 93%.

Exposing concrete structures to severe environmental conditions causesdamage and reduction in the service life of structural members, such asbridge columns, bridge decks, bridge superstructures, and buildings(ASCE, (2017), Infrastructure report card. Reston, Va.: ASCE; which ishereby incorporated herein by reference in its entirety). Repairing andretrofitting these structural elements by applying repair materials,such as normal strength concrete, polymer concrete, and ultra-highperformance concrete (Valikhani, et al., 2018, Experimentalinvestigation of high-performing protective shell used for retrofittingbridge elements (No. 18-05142), Florida International University; whichis hereby incorporated herein by reference in its entirety), to concretesubstrates can be an economical option compared to the replacement ofthe entire structure. The bond strength between old concrete substratesand new repair materials plays a vital role in the selection ofappropriate repair materials. In order to enhance bond strengthcharacteristics between concrete substrate and repair material,roughening substrate surfaces with different techniques, such assandblasting and water jetting may be preferable compared to grinding orwire brushing or chipping techniques. Additional advantages of bothsandblasting and water jetting are attributed to the removal of largeareas of the damaged substrate in a short time, preparing a sufficientroughness for the surface, avoiding micro-cracks, and introducing thehighest bond strength. A drawback of these techniques is related to thevariation of surface finish based on tools, technician experience, ageof materials, and time of processing. After the removal of the damagedconcrete, the substrate roughness is usually assessed based onqualitative methods and observation, which cannot be relied upon as aneffective tool to quantitatively evaluate surface preparation due tofactors including qualitative judgement, technician experience, andhuman errors.

The International Concrete Repair Institute (ICRI, (1997), Selecting andspecifying concrete surface preparation for sealers, Coatings andPolymer Overlays, No. 03732, Des Plaines, Ill.: ICRI; which is herebyincorporated herein by reference in its entirety), proposed 10 differentconcrete surface profiles, which provide visual standards for fast andeasy inspection and evaluation. Nevertheless, the results of this methodare simply qualitative and subjected to technician judgment and can bealtered from one technician to another due to human errors. Inspecifications, to address the interfacial bond strength, the substratesurface may be categorized based on surface finishing treatments, andcoefficients of cohesion and friction may be calculated based on limitedcategories without direct correlation between surface roughness andsurface parameters. The American Concrete Institute of 2008 (Buildingcode requirements for structural concrete (ACI 318-08) and commentary,Farmington Hills, Mich.: ACI, ISBN: 978-0-87031-264-9; which is herebyincorporated herein by reference in its entirety) categorizes concretesurface roughness into four groups as concrete placed against cleanconcrete, surface with intentional roughness of amplitude of 6 mm (0.25in.), concrete cast monolithically, and concrete placed next tostructural steel section, whereas in AASHTO-LRFD (Bridge designspecifications, American Association of State Highway and TransportationOfficials, 5-138, 2017; which is hereby incorporated herein by referencein its entirety), two additional categories for lightweight concrete andcast-in-place concrete slab on clean concrete girder surfaces arediscussed. In the fib model (Beton, D. (2013), fib model code forconcrete structures 2010, Wiley-VCH Verlag Gmbh; which is herebyincorporated herein by reference in its entirety), four categories aredefined from very smooth to very rough based on surface roughness.Although this categorization brings ease of use by designers, it doesnot effectively correlate the degree of surface roughness to surfaceparameters and bond strength. Therefore, quantitative methods to measurethe concrete surface roughness are required to minimize humanintervention.

Many quantitative methods have been developed in the literature,including numerous physical, digital, sensor based, optical,photographic, and microscopic methods. However, 3D laser scanning (see,e.g., Santos, et al., (2010), Comparison of methods for textureassessment of concrete surfaces, ACI Materials Journal, 107(5), 433-440;which is hereby incorporated herein by reference in its entirety) isidentified as a suitable quantitative method as described below.

Embodiments provide novel noncontact methods for industrial applicationto evaluate surface roughness of concrete substrate using digital imageprocessing and machine learning via basic cameras (e.g., smartphonecameras, tablet cameras, or consumer grade digital cameras.) Forexample, any device (e.g., a smartphone) capable of capturing images(e.g., images of at least 12 megapixels) can be used. The providedindustrial application methods have been compared to a well-establishedmethod using 3D laser scanning. In a first novel method, digital imageprocessing is utilized by introducing a new index (AR) using digitalimage processing to separate the aggregates from cement paste, and thentaking a ratio of the aggregate area to total area which is calculatedand correlated to surface roughness as obtained from 3D laser scanning.Since the provided image processing method is independent from othermethods, the results may depend on classifying the new images to acertain class resulting in a robust qualitative method. To quantify theresults, the index AR can be correlated to a degree of surface roughnesssuch as obtained from laser scanner or any other methods.

In a second novel method, a practical application utilizing machinelearning is provided. A deep neural network, also known as deeplearning, has been utilized to extract deep and meaningful featuresdirectly from raw images as an automated method without any manualfeature extraction and human efforts. The benefits of this methodinclude: minimizing human intervention and minimizing errors related toenvironmental conditions, camera angle, and camera configuration. Theuse of these methods can offer other industrial alternatives to the 3Dlaser scanner, which is relatively high in cost and not widely availableif compared to the use of basic cameras or smartphones. The bondstrength between concrete substrate, prepared using sandblasting, andrepair material can be calculated based on concrete surface roughness(see, e.g., Santos, et al., 2007, Correlation between concrete-toconcrete bond strength and the roughness of the substrate surface,Construction and Building Materials, 21(8), 1688-1695; see also, e.g.,Valikhani, et al., 2020, Experimental evaluation of concrete-to-UHPCbond strength with correlation to surface roughness for repairapplication, Construction and Building Materials, 238,https://doi.org/10.1016/j.conbuildmat.2019.117753; each of which ishereby incorporated by reference herein in its respective entirety).

Retrofitting and rehabilitation of concrete elements (e.g., by casting anew layer of ultra-high-performance concrete (UHPC)) may require a goodbond between old and new layers to guarantee the success of this newmethod of retrofitting. It has been shown that the bond interfacebetween two layers of concrete directly links to the surface roughnessof the substrate (Valikhani, et al., Experimental evaluation ofconcrete-to-UHPC bond strength with correlation to surface roughness forrepair application, Construction and Building Materials 238 (2020):117753; which is hereby incorporated herein by reference in itsentirety).

In order to roughen the substrate surface, sandblasting technology maybe used. Embodiments of the subject invention provide methods toevaluate the roughness of concrete surfaces. Concrete surface roughnessevaluation can be used in retrofitting and rehabilitation of damagedconcrete elements or casting a new layer of concrete over a precastconcrete.

Embodiments provide novel non-contact methods to evaluate concretesurface roughness from images by leveraging digital image processing ormachine learning techniques. Digital image processing may be used todevelop a novel concrete roughness index (AR) which is calculated as theratio between the total area of exposed coarse aggregates and totalconcrete surface by separating aggregates from cement paste. Forexample, AR of zero or close to zero represents a very smooth finish ofthe concrete surface, whereas AR of 1 represents only coarse aggregate.FIG. 13 shows three different surface roughness with corresponding AR.To maximize computational efficiency, the novel concrete roughness index(AR) may be used to label and create classes for images for training aconvolution neural network (i.e., machine learning) by leveraging dataaugmentation and transfer learning techniques to utilize a relatively asmall dataset (e.g., tens or hundreds of surface roughness images may beused to train the model, instead of thousands or tens of thousands.)

Embodiments provide novel application methods for estimating concretesurface roughness which have been related to and compared against awell-established method wherein 3D laser scanning is utilized to measurethe concrete surface roughness for nine cubic specimens with a surfacearea of 153 mm×153 mm (6 in.×6 in.), which were roughened usingsandblasting. In the first method, digital image processing is utilizedto find the correlation between aggregate area and surface roughness. Inthe second method, an advanced machine learning technique is introducedto classify and calculate the surface roughness. Finally, to evaluatethe efficiency of both methods, the surface roughness of a large scaleT-beam specimen is assessed, and the results are compared to the 3Dlaser scanning method, independently.

A 3D laser scanner method for estimating concrete surface roughness(Santos, et al., 2010) provides several advantages, such as ease ofapplication, high measurement accuracy, noncontact method ofmeasurement, and less sensitivity to the distance or angle of scanning;but comes with a high cost, complexity, and limited availability

To measure concrete surface roughness using a 3D or 2D laser scanner,several steps should be conducted. In the first step, the concretesurface should be scanned to obtain a 2D image of the surface, as shownin FIG. 1A. In the second step, the scanned image of the concretesurface may be imported into a software (e.g., MATLAB) to generate a 3Dcoordinate surface based on the peaks and valleys of the surface. In thethird step, an optimized smooth plane should be obtained to equalize thearea of the peaks to the area of the valleys, as shown in FIG. 1B in 2Drepresentation. In the final step, the degree of concrete surfaceroughness (Ra) should be calculated using Equation (1), which ismodified from Santos, et al., (2010) by utilizing a discrete domaininstead of a continuous domain by integrating the absolute peaks andvalleys along the length of the two sides of the surface. One drawbackof 3D laser scanning is that this method cannot distinguish whether thepeaks and valleys are made of aggregate or cement.

$\begin{matrix}{R_{a} = \frac{\sum{y_{i}}}{n}} & (1)\end{matrix}$where Ra is average concrete surface roughness; |y_(i)| is absolutedistance of the peaks and valleys from the optimized plane, as shown inFIG. 1B, and n is number of data points associated with the scannedsurface.

This method was used to measure the concrete surface roughness of ninesmall specimens and results were utilized to relate and calibrate bothdigital image processing and machine learning methods, respectively,against the 3D laser scanner results in Example 1, below.

According to an embodiment of the subject invention, several images weretaken for each of the nine samples using a commercial camera of asmartphone with a quality of 12 megapixels. Based on these images, itwas observed that the coarse aggregates were lighter in color comparedto cement paste, and this variation in color can be used as an index tomeasure concrete surface roughness. In much rougher surfaces, the numberof exposed coarse aggregates increases. The digital image processingmethod is sensitive to environmental conditions, such as lighting, dust,and darkness. Therefore, consistency in the condition of obtaining theimages may be extremely important in some embodiments. To be consistentand to reduce the potential effect of these issues, the images weretaken in a dark room with flash applied.

A binary decision was applied for pixel intensity of 0 for cement paste(black) and 1 for the aggregates (white) using Equation (2) to form abinary image (g(n)) per segment.

$\begin{matrix}{{g(n)} = \left\{ \begin{matrix}{{1\mspace{14mu}{if}\mspace{14mu}{k(n)}} < {T\mspace{14mu}{for}\mspace{14mu}{white}\mspace{14mu}{pixel}}} \\{{0\mspace{14mu}{if}\mspace{14mu}{k(n)}} < {T\mspace{14mu}{for}\mspace{14mu}{black}\mspace{14mu}{pixel}}}\end{matrix} \right.} & (2)\end{matrix}$

The number of white pixels, representing the aggregate (Equation 3) andthe total number of white and black pixels, representing both aggregatesand cement (Equation 4), were calculated for each image segment. Thetotal aggregate area was then calculated for the whole image from eachsegment (Equation 5) by multiplying the number of white pixels times thearea of each pixel. The total surface area was then calculated for thewhole image from each segment (Equation 6) by multiplying the number ofblack pixels and white pixels times the area of each pixel. The ratiobetween the total aggregate area and surface area, AR, was used as anindex of concrete surface roughness (Equation 7). The AR then can berelated to the surface roughness calculated from the 3D laser scanningmethod with a function based on the sample results.

$\begin{matrix}{{{number}\mspace{14mu}{of}\mspace{14mu}{white}\mspace{14mu}{pixels}\mspace{14mu}{for}\mspace{14mu}{each}\mspace{14mu}{segment}} = {{n({segment})} = {\sum\limits_{1}^{W}{\sum\limits_{1}^{H}\left\lbrack {K(1)} \right\rbrack}}}} & (3) \\{{{Number}\mspace{14mu}{of}\mspace{14mu}{black}\mspace{14mu}{and}\mspace{14mu}{white}\mspace{14mu}{pixels}\mspace{14mu}{for}\mspace{14mu}{each}\mspace{14mu}{segment}} = {{N({segment})} = {\sum\limits_{1}^{W}{\sum\limits_{1}^{H}\left\lbrack {{K(0)} + {K(1)}} \right\rbrack}}}} & (4)\end{matrix}$where H is a vertical coordinate in the segment (e.g., H=150 pixels inExample 1); W is a horizontal coordinate in the segment (e.g., W=150pixels in Example 1); K(0) is a black pixel (digit 0) and K(1) is awhite pixel (digit 1).

The total area of aggregate (A_(aggregate)) for each image is given byEquation (5).

$\begin{matrix}{A_{aggregate} = {\sum\limits_{1}^{s{egment}}{{n({segment})} \times A_{pixel}}}} & (5)\end{matrix}$

The total area of surface (A_(surface)), including aggregate pluscement, for each image is given by Equation (6).

$\begin{matrix}{A_{surface} = {\sum\limits_{1}^{s{egment}}{{N({segment})} \times A_{pixel}}}} & (6)\end{matrix}$where, e.g., Apixel=(0.254 mm)²=(0.01 in)² and segment=16, in Example 1.AR=A _(aggregate) /A _(surface)  (7)

Machine learning is a set of techniques that learn and experience fromdata without implementing an explicit program. Machine learning has beenutilized in structural, civil, construction, bridge engineering,pavement, inspection, structural health monitoring (SHM), and earthquakeengineering.

In other fields, deep learning and convolutional neural networks (CNN)have been used in various computer vision applications including thermalinfrared face identification, vehicle classification in trafficsequences, image super resolution techniques, as well as an ensemble ofCNNs to overcome the low resolution images of surveillance cameras forvehicle type recognition, and atrous convolutions and spatial pyramidsfor pupil detection and eye tracking.

Embodiments may provide a combination of advanced techniques in machinelearning and computer vision including transfer learning and dataaugmentation which may be utilized to facilitate the use of smalldatasets in training for more practical industrial applicationsincluding evaluation of concrete surface roughness. The introduction oftransfer learning and data augmentation may save the cost and effort ofprocessing thousands or even millions of raw images with ground truthinformation (e.g., identification of corresponding degrees of surfaceroughness using 3D laser scanning). This increase in training efficiencymay provide real technical and commercial benefits across many of theapplication areas mentioned above.

The great success of machine learning and deep learning techniques inthe area of image and visual data processing heavily depends on theavailability of large scale annotated datasets to learn the existingpattern in the data. However, collecting large scale image dataset withlabels may be time consuming, tedious, and expensive, especially givenissues of access and difficulty in obtaining quality images in manyconcrete repair applications (e.g., bridges, buildings, or highways areoften remote, may be either elevated or covered by ground or water, andmay be difficult to reach.) Embodiments of the subject invention providea powerful technique called data augmentation (see, for example,Pouyanfar, et al., (2019), Unconstrained flood event detection usingadversarial data augmentation, 2019 IEEE International Conference onImage Processing (ICIP) (p. 155); which is hereby incorporated herein byreference in its entirety) to generate synthetic training images fromexisting data. This method helps the deep learning model to begeneralized to new conditions and environments. If the model only trainson the current image dataset, it may be difficult or almost impossibleto predict the class of new images (e.g., images created under differentconditions.)

Using data augmentation according to embodiments of the subjectinvention, it is possible to generate a variety of new samples bychanging different characteristics of existing images. Embodimentsprovide augmentation operations including random rotation, blur,brightness, horizontal and vertical flipping, and resizing the images.Considering the fact that deep neural networks may require thousands oreven millions of data samples with labels to learn the parameters of thenetwork, it is still difficult to train a deep learning model on such asmall dataset (e.g., less than a dozen images, dozens of images, orhundreds of images.) Thus, it may be advantageous to leverage existingpretrained models and transfer the knowledge from a large dataset (e.g.,recognizing edges, shapes, and other common features) to this smalldataset. This technique is called transfer learning. In other words,knowledge learned from a large scale dataset, such as ImageNet (ImageNet(2019), image-net, Retrieved from http://www.image-net.org; which ishereby incorporated by reference herein in its entirety), can betransferred to the proposed domain with a small number of images. Thetransfer learning technique has several advantages. First, it reducesthe necessity of having large labeled training datasets, which is verytime consuming and needs lots of human efforts. Second, it enhances themodel performance due to the extra knowledge it captures from a sourcedomain, which may not be available in the target problem. Moreover, itcan significantly reduce the training time of the model development byutilizing the knowledge from a source domain instead of learning a modelfrom scratch. (See, e.g., Yang, et al., (2019), Multi-object trackingwith discriminant correlation filter based deep learning tracker,Integrated Computer-Aided Engineering, 26(3), 273-284; which is herebyincorporated by reference herein in its entirety; see also, e.g., Gao,et al., (2018), Deep transfer learning for image based structural damagerecognition, Computer-Aided Civil and Infrastructure Engineering, 33(9),748-768; which is hereby incorporated by reference herein in itsentirety).

Turning now to the figures, FIG. 1A is an image of a concrete surfaceroughened by sandblasting, suitable for analysis by systems or methodsaccording to embodiments of the subject invention. FIG. 1B is atwo-dimensional representation of a scanned surface with peaks andvalleys and a best fit plane produced from a 3D laser scanning method.

FIGS. 2A through 2F illustrate a digital image processing sequenceaccording to an embodiment of the subject invention. FIG. 2A is a fullcolor RGB image of a concrete sample ready for image processingaccording to an embodiment of the subject invention. This image may beobtained with a commercially available imaging system (e.g., a consumergrade camera, smartphone, or tablet). FIG. 2B is an RGB image afterapplying a contrast enhancement filter for image processing according toan embodiment of the subject invention.

The digital image processing may be implemented in a commerciallyavailable analysis Software (e.g., MATLAB) or through custom softwaremodules. The digital image processing may be implemented as a mobileapplication for use in the field (e.g., on a mobile device app directlyat the site of image collection, or on a laptop computer or server at ajobsite trailer or mobile operations center) or at a remote location(e.g., on a laptop, desktop, or server at a remote data center, office,or cloud computing environment.)

FIG. 2C is a segmentation of the RGB image after applying contrastenhancement filter for image processing according to an embodiment ofthe subject invention. Images may be segmented in a 4×4 grid as shown,producing 16 segments for analysis; alternatively in a 2×2, 3×3, 5×5,6×6, 7×7, 8×8, 9×9, or larger grid, including increments, combinations,and ranges of any of the foregoing. Segmentation may be symmetric (e.g.,N×N, where N is an integer) or asymmetric (e.g., N×M where N and M areintegers and N M.) Segmentation size may be determined by fixed values(e.g., based on image size; available computing resources such asmemory, storage, or processing power; or expected or known sampleparameters such as aggregate size, aggregate distribution, estimated orexpected roughness, or reference measurements taken before, concurrentwith, or after image collection.) Segmentation may be evenly distributed(e.g., all segments the same size), unevenly distributed (e.g., somesegments larger or smaller than others), or adaptively distributed(e.g., segment size chosen by an algorithm, formula, iteration,analysis, or human intervention) based on image properties (e.g.,number, size, or distribution of aggregate; or initial digital imageanalysis results.)

FIG. 2D is a segmented gray-scale image for image processing accordingto an embodiment of the subject invention. FIG. 2E is a segmented imagein the process of threshold selection for image processing according toan embodiment of the subject invention. The crosses in FIG. 2E representthe points that were manually selected to separate the aggregate fromthe cement. FIG. 2F is a black and white image showing separation ofaggregate for image processing according to an embodiment of the subjectinvention. Gray-scale conversion, threshold selection, and black andwhite mapping may be performed with or without human intervention andwith or without automated analysis, algorithms, or computer intervention(e.g., including digital image processing, deep learning, machinelearning, or other approaches.)

FIG. 3A is an image of concrete surface in office light used inaccordance with embodiments of the subject invention. FIG. 3B is animage of concrete surface in a dark room with flash used in accordancewith embodiments of the subject invention. FIG. 3C is an image ofconcrete surface from a scanner used in accordance with embodiments ofthe subject invention. FIG. 3D is a scanned image of concrete surfacefrom an angle used in accordance with embodiments of the subjectinvention. Embodiments of the subject invention may provide processingto account for images taken from the above and other input conditions(e.g., including natural light, daylight, night lighting, ambientlighting, street lights, and flash applied with or without daylight,natural light, full sun, shade, night-time lighting, and with or withouta screen, box, blind, hood, tarp, or other method of blocking, reducing,directing, or controlling existing or additional light sources.

FIG. 4A is a sample of an original image of a concrete surface used inaccordance with embodiments of the subject invention. FIG. 4B is asample of an augmented flipped original image of a concrete surface usedin accordance with embodiments of the subject invention. FIG. 4C is asample of an augmented flipped image of a concrete surface used inaccordance with embodiments of the subject invention. FIG. 4D is asample of an augmented blurry image of a concrete surface used inaccordance with embodiments of the subject invention. Augmentation ofimages according to embodiments of the subject invention may include theabove and other manipulations known in the art.

FIG. 5 is a block diagram of the ResNet model used in accordance withembodiments of the subject invention. In addition to the model shown inFIG. 5, various additional models, summing or other functions, andformulations for F(x) known in the art or later discovered, published,or developed, are contemplated for use with embodiments.

FIGS. 6A through 6I show sample images 1 through 9, respectively, asused to test a digital image processing method and a machine learningmethod according to embodiments of the subject invention, and as laterclassified in FIG. 14.

FIG. 7 is a chart showing a correlation between the ratio of theaggregate area-to-total area (AR) to concrete surface roughness (Ra)according to an embodiment of the subject invention. The larger discretepoints represent individual data points and the smaller dotted linerepresents Equation 9 as fit to this data.

FIG. 8 is a chart showing the accuracy of augmented (solid line) andnon-augmented (dashed line) deep learning models at each epoch accordingto an embodiment of the subject invention. The of augmented (solid line)model approaches an accuracy of about 0.9 after about 60 epochs. Thenon-augmented (dashed line) model approaches an accuracy of about 0.4after about 93 epochs. FIG. 8 is related to the training processdepicted in FIGS. 3A-3D and 4A-4D, and FIG. 9 is related to samplesidentified in FIGS. 6A-6I.

FIG. 9 is a chart showing the loss values of augmented (solid line) andnon-augmented (dashed line) deep learning models at each epoch accordingto an embodiment of the subject invention. The augmented (solid line)model approaches a loss of about 0.25 after about 80 epochs. Thenon-augmented (dashed line) model approaches a loss of between about 1.5and about 2.0 after about 40 epochs.

FIG. 10 is an illustration of a large scale (H=419 mm; L=4,724 mm;X=203.2 mm; Y=203.2 mm) T-beam with roughened surface with six samplesmarked for analysis according to an embodiment of the subject invention.

FIGS. 11A through 11F show large-scale sample images 1 through 6extracted from the beam represented in FIG. 10 and used to test adigital image processing method and a machine learning method accordingto embodiments of the subject invention. Six images were used in thevalidation of both digital image processing and machine learning systemsand methods. The images were taken from roughened areas on a web of alarge T-beam with a total length of 4,724 mm (186 in.) and a height of419 mm (16.5 in.). Each sample was 203.2 mm×203.2 mm (8 in.×8 in.) insize.

FIG. 12 is a chart showing a comparison between a 3D laser scanningmethod (solid line), an image processing method according to anembodiment of the subject invention (filled circles), and a machinelearning method (filled triangles) according to an embodiment of thesubject invention. The dashed lines represent +/−10% variance from the3D laser scanning method. The horizontal axes represent Ra from the 3Dlaser scanning method in inches and millimeters, respectively. Thevertical axes represent Ra calculated according to the subject inventionin inches and millimeters, respectively.

FIG. 13 shows a value of concrete surface roughness index AR for each ofthree different concrete surfaces measured according to an embodiment ofthe subject invention. Smooth concrete with no aggregate as shown in theleft most image produces an AR of zero. Sandblasted concrete with someaggregate exposed produces an AR of 0.3232. Sandblasted concrete withmore aggregate exposed produces an AR of 0.4509. In FIG. 13, the middleimage matches FIG. 6D (rotated 90 degrees), and the right-hand imagematches FIG. 6A (rotated 90 degrees).

FIG. 14 shows digital image processing results, AR, used to classify andlabel images from FIGS. 6A through 6I into three categories for themachine learning training phase according to an embodiment of thesubject invention. The images from FIGS. 6A and 6B were grouped intoclass C1, with the highest roughness and had an average AR of 0.4428.The images from FIGS. 6C, 6D, 6E, and 6F were grouped into class C2,with the medium roughness and had an average AR of 0.3380. The imagesfrom FIGS. 6G, 6H, and 6I were grouped into class C3, with the lowestroughness and had an average AR of 0.2086.

FIG. 15 shows a detailed confusion matrix for a deep learning model withtransfer learning and data augmentation (“aug.” model) according to anembodiment of the subject invention. Additional data is provided for onesample from “Real Class” C1 (i.e., as determined by 3D laser scannermethod) and “Predicted Class” C2 (i.e., as determined by digital imageprocessing and augmented machine learning methods according to anembodiment of the subject invention.) This image was classified in C2 bythe aug. deep learning model with a P1 probability value of 0.42 (i.e.,42% chance of being in Class C1), a P2 probability value of 0.55 (i.e.,55% chance of being in Class C2), a P3 probability value of 0.03 (i.e.,3% chance of being in Class C3).

FIG. 16 shows a detailed confusion matrix for deep learning model withtransfer learning and without augmentation (“non-aug.” model) accordingto an embodiment of the subject invention. Additional data is providedfor three samples from “Real Class” C1 (i.e., as determined by 3D laserscanner method) and “Predicted Class” C3 (i.e., as determined by digitalimage processing and non-augmented machine learning methods according toan embodiment of the subject invention.) For example, the first image atthe top of the C3 column was classified in C3 by the non-aug. deeplearning model with a P1 probability value of 0.27 (i.e., 27% chance ofbeing in Class C1), a P2 probability value of 0.06 (i.e., 6% chance ofbeing in Class C2), a P3 probability value of 0.67 (i.e., 67% chance ofbeing in Class C3). Additional data is also provided for six samplesfrom “Real Class” C2 (i.e., as determined by 3D laser scanner method)and “Predicted Class” C3 (i.e., as determined by digital imageprocessing and non-augmented machine learning methods according to anembodiment of the subject invention.) For example, the first image atthe top of the cell at the intersection of the C2 row and C3 column wasclassified in C3 by the non-aug. deep learning model with a P1probability value of 0.27 (i.e., 27% chance of being in Class C1), a P2probability value of 0.12 (i.e., 12% chance of being in Class C2), a P3probability value of 0.61 (i.e., 61% chance of being in Class C3).

The improved results of the augmented model can be seen by comparingFIG. 15 to FIG. 16, as the augmented model agreed more closely with the3D laser scanning results.

Embodiments of the subject invention address the technical problem ofestimating the degree of concrete surface roughness being expensive,needing excessive human processing, not being suitable for assessing oldstructures based on their inspection records, and requiring contactmethods. This problem is addressed by providing digital image processingwith a new index for concrete surface roughness based on the aggregatearea-to-total surface area (AR), in which a machine learning methodapplying a combination of advanced techniques is utilized to categorizeimages based on the classification given during the learning process.

The transitional term “comprising,” “comprises,” or “comprise” isinclusive or open-ended and does not exclude additional, unrecitedelements or method steps. By contrast, the transitional phrase“consisting of” excludes any element, step, or ingredient not specifiedin the claim. The phrases “consisting” or “consists essentially of”indicate that the claim encompasses embodiments containing the specifiedmaterials or steps and those that do not materially affect the basic andnovel characteristic(s) of the claim. Use of the term “comprising”contemplates other embodiments that “consist” or “consisting essentiallyof” the recited component(s).

When ranges are used herein, such as for dose ranges, combinations andsubcombinations of ranges (e.g., subranges within the disclosed range),specific embodiments therein are intended to be explicitly included.When the term “about” is used herein, in conjunction with a numericalvalue, it is understood that the value can be in a range of 95% of thevalue to 105% of the value, i.e. the value can be +/−5% of the statedvalue. For example, “about 1 kg” means from 0.95 kg to 1.05 kg.

The methods and processes described herein can be embodied as codeand/or data. The software code and data described herein can be storedon one or more machine-readable media (e.g., computer-readable media),which may include any device or medium that can store code and/or datafor use by a computer system. When a computer system and/or processorreads and executes the code and/or data stored on a computer-readablemedium, the computer system and/or processor performs the methods andprocesses embodied as data structures and code stored within thecomputer-readable storage medium.

It should be appreciated by those skilled in the art thatcomputer-readable media include removable and non-removablestructures/devices that can be used for storage of information, such ascomputer-readable instructions, data structures, program modules, andother data used by a computing system/environment. A computer-readablemedium includes, but is not limited to, volatile memory such as randomaccess memories (RAM, DRAM, SRAM); and non-volatile memory such as flashmemory, various read-only-memories (ROM, PROM, EPROM, EEPROM), magneticand ferromagnetic/ferroelectric memories (MRAM, FeRAM), and magnetic andoptical storage devices (hard drives, magnetic tape, CDs, DVDs); networkdevices; or other media now known or later developed that are capable ofstoring computer-readable information/data. Computer-readable mediashould not be construed or interpreted to include any propagatingsignals. A computer-readable medium of embodiments of the subjectinvention can be, for example, a compact disc (CD), digital video disc(DVD), flash memory device, volatile memory, or a hard disk drive (HDD),such as an external HDD or the HDD of a computing device, thoughembodiments are not limited thereto. A computing device can be, forexample, a laptop computer, desktop computer, server, cell phone, ortablet, though embodiments are not limited thereto.

A greater understanding of the embodiments of the subject invention andof their many advantages may be had from the following examples, givenby way of illustration. The following examples are illustrative of someof the methods, applications, embodiments, and variants of the presentinvention. They are, of course, not to be considered as limiting theinvention. Numerous changes and modifications can be made with respectto embodiments of the invention.

Example 1: Digital Image Analysis

A 3D laser scanner method for estimating concrete surface roughness(Santos, et al., 2010) was used to measure the concrete surfaceroughness of nine small specimens. Several images were taken for each ofthe nine samples in a dark room with flash applied using a commercialcamera of a smartphone with a quality of 12 megapixels.

After obtaining high quality images for the concrete surfaces, theresolution was enhanced to 600×600 pixels from an original resolution of300×300 pixels for the small surfaces with an area size of 153 mm×153 mm(6 in.×6 in.), as shown in FIG. 2A.

The images were analyzed in MATLAB software in five stages:

1) A contrast enhancement filter was applied to all images to betterdistinguish boundary intensity between aggregate and cement, as shown inFIG. 2B.

2) To increase the accuracy of the analysis, each image was divided into16 equal size segments, as shown in FIG. 2C.

3) The true color (RGB) image was transferred into a grayscale image, asshown in FIG. 2D, with pixel values k(n) ranging from 0 to 255. For eachsegment within the image, pixels representing aggregates were chosenmanually and the pixel values k(n) were averaged to determine thethreshold value (T), as shown in FIG. 2E. After applying the threshold,the accuracy of the method is checked qualitatively in an iterativemanner.

4) The grayscale image was transferred to the black and white image, asshown in FIG. 2F, with a binary decision for pixel intensity of 0 forcement paste (black) and 1 for the aggregates (white) using Equation (2)forming a binary image g(n) per segment.

5) The number of white pixels, n(segment), representing the aggregate(Equation 3) and the total number of white and black pixels, N(segment),representing both aggregates and cement (Equation 4), were calculatedfor each image segment. With a resolution of 150×150 pixels, where H isvertical coordinate in the segment (H=150 pixels in this example); W ishorizontal coordinate in the segment (W=150 pixels in this example);K(0) is a black pixel (digit 0) and K(1) is a white pixel (digit 1). Thetotal aggregate area, A_(aggregate), was then calculated for the wholeimage from each segment (Equation 5) by multiplying the number of whitepixels times the area of each pixel. The total surface area,A_(surface), was then calculated for the whole image from each segment(Equation 6) by multiplying the number of black pixels and white pixelstimes the area of each pixel. Where A_(pixel)=(0.254 mm)2=(0.01 in)2 andsegment=16, in this example. The ratio between the total aggregate areaand surface area, AR, was used as an index of concrete surface roughness(Equation 7).

The resulting AR then can be related to the surface roughness calculatedfrom the 3D laser scanning method with a function based on the sampleresults.

Example 2A: Machine Learning Introduction

Machine learning and deep learning techniques may be employed accordingto embodiments of the subject invention to eliminate human errors,provide an automated and practical tool for estimating concrete surfaceroughness, and assess old structures based on images from priorinspection records. The drawbacks of digital image processing are timeconsumption, extensive human intervention, and the sensitivity toenvironmental conditions, such as type of camera used for imaging,object distance, and angle from the camera. FIGS. 3A through 3D showimages obtained for the same surface with different conditions ofimaging.

The provided machine learning method may be defined as a classificationtechnique that predicts the class of each image based on its deep visualfeatures. Classification is a supervised machine learning method thatneeds data with preassigned labels. The model can learn from theexisting data samples to predict future labels. In this study, labelsare defined as levels of surface roughness, which may be obtained from3D laser scanning for comparison. Images may be divided into trainingand testing sets. The training set includes the data that are used fortraining the machine learning model, while the testing set includes thedata for validating the model. In an embodiment, 78-88% of images areused for training and 22-12% of them are utilized for testing.Alternatively, 70, 75, 80, 85, 90, or 95% of available images may beused for training, including increments, combinations, and ranges of anyof the foregoing. In certain commercial embodiments, some fraction orall available images including customer images, research images, publicdatabase images, existing images, or specially collected images may beused to train the network before using the system In order toautomatically generate a variety of images to simulate additional imageswith real conditions, such as different lights, image source, imagesize, rotation, and so on, various data augmentation techniques may beapplied, including random left and right rotation (e.g., rotation of25-degrees, alternatively 5, 10, 15, 20, 30, 35, 40, or 45-degrees,including increments, combinations, and ranges of any of the foregoing),change in brightness, blur with a uniform filter (e.g., of size 11),horizontal, angled, and vertical flipping, and cropping (or evenresizing) the images. Data augmentation is applied in an offline mannerto increase the sample sizes of training data. These operations areapplied only on the training data and increase the sample size. Thelighting parameter can be, for example, any number from 0 to 1 (e.g., avalue of 0.2 can reduce or increase the brightness by 20%). Image sizecan be fixed, and a factor (e.g., 0.2) can be used for zoon in and/orzoom out. A blur size of, for example, 11 can be used, thoughembodiments are not limited thereto. The blur size can be other values(e.g., 7), and these number can be adjusted and don't need to be fixed.

In this example nine small specimens were categorized into threeclasses, as described herein, based on the surface roughness degree from3D laser scanning. Images of these nine specimens were augmented toincrease the number of samples available for training. Exemplary samplesof the augmentation results are shown in FIGS. 4A through 4D. In total,each class contains 50 training images, including both real andartificially generated images (total of 150 images across 3 classes; 63real images and 87 artificially generated images).

In certain embodiments (e.g., as in Example 2), a pretrained model suchas ResNet50 (He, et al., (2016), Deep residual learning for imagerecognition, Proceedings of the IEEE Conference on Computer Vision andPattern Recognition (pp. 770-778); which is hereby incorporated hereinby reference in its entirety) may be leveraged to extract the deepvisual features from the data automatically. Other models may be usedwith embodiments and are contemplated within the scope of the subjectinvention. ResNet50 is a 50-layer CNN with residual connections, whichavoids vanishing gradients and enhances the model accuracy. A sample ofthe residual diagram is shown in FIG. 5, where (X) is the input of eachlayer, and F(X) is the output. The original input is added to the outputof the convolution block to produce F(X)+X. This connection is calledskip (residual). Skip connections allow layers to skip layers andconnect to layers further up the network, allowing for information toflow more easily up the network. The family of models can be, forexample, a convolutional neural network (CNN), and ResNet50 is a commonmethod of CNN that can be used for feature extraction.

Example 2B: Machine Learning Analysis

A set of nine small specimens were categorized into three classes basedon the surface roughness degree measured by 3D laser scanning of eachspecimen. As represented in FIG. 14, the samples are separated intothree classes (C1, C2, and C3) representing the roughest surface to thesmoothest. In this technique, when a brand new image is given to thenetwork, the output of the network generates three probabilities (e.g.,[P1, P2, P3]) which represents the probability of matching with classesfrom C1, C2, or C3, respectively).

Advantages of this method include finding the predicted class by gettingthe maximum probability and estimating the roughness of the surface bygetting a weighted average of the roughness of each class. Morespecifically, the roughness may be calculated using Equation 8A.

Embodiments provide many advantages, among them are the following.

1. Inexpensive method compared to other methods such as the 3D laserscanner.

2. Accuracy up to 93% of results obtained from 3D laser scanner.

3. Possible deployment in mobile (e.g., smart phone or tablet app) withmachine learning

4. Reducing or eliminating human intervention using deployed app (e.g.,in smart phones, tablet, mobile, edge computing, or cloud-based device)

5. Assisting engineers in evaluating concrete surface roughness forretrofitting and repair of concrete structures.

6. Computationally inexpensive by utilizing digital image processing fordeveloping index and labelling and machine learning for classification

Images were augmented to increase the number of samples available fortraining. Examples of the augmentation results are shown in FIGS. 4Athrough 4D. In total, each class contains 50 training images, includingboth real and artificially generated images (total of 150 images across3 classes; 63 real images and 87 artificially generated images).

A pretrained model, ResNet50 in this example, was leveraged to extractthe deep visual features from the data automatically. ResNet50 is a50-layer CNN with residual connections, which avoids vanishing gradientsand enhances the model accuracy. A sample of the residual diagram isshown in FIG. 5, where (X) is the input of each layer, and F(X) is theoutput. The original input is added to the output of the convolutionblock to produce F(X)+X. This connection is called skiper (residual).

In this example the model is pretrained on the ImageNet dataset (a verylarge scale image dataset with millions of samples and 1,000 classes).The last layer of this model is removed, and a new classification layer(also called softmax) with three outputs is added to this network topredict the corresponding classes. The weight of this layer isinitialized randomly to be learned by the new dataset, while all otherlayers' weights are frozen, which means that the layer weights are notupdated during the training. In other words, the filters in the firstconvolutional layers in ResNet have already learned to recognize edges,colors, angles, circles, and so on and can be directly used to extractgeneral features from images without needing a large scale dataset.)ImageNet is the largest dataset used for all these computer visionclassification tasks and all these models are only pretrained onImageNet, if the task was object detection or segmentation there wasother datasets such as COCO or PASCAL. ImageNet is only provided as oneexample and should not be construed as limiting. The softmax function,also known as softargmax or normalized exponential function, is ageneralization of the logistic function to multiple dimensions. It isused in multinomial logistic regression and is often used as the lastactivation function of a neural network to normalize the output of anetwork to a probability distribution over predicted output classes,based on Luce's choice axiom. The softmax function takes as input avector z of K real numbers, and normalizes it into a probabilitydistribution consisting of K probabilities proportional to theexponentials of the input numbers.

Based on the result of 3D laser scanning, the samples are separated intothree classes (C1, C2, and C3) representing the roughest surface to thesmoothest. In this technique, when a brand new image is given to thenetwork, the output of the network generates three probabilities (e.g.,[P1, P2, P3]), which represents the probability of matching with classesC1, C2, C3, respectively. This method has at least two notable benefits;first, it is possible to find the predicted class by getting the maximumobvious probability and second, it can estimate the roughness of thesurface (e.g., by getting a weighted average of the roughness of eachclass multiplied by the respective probability of matching that class.)The roughness may be estimated from the model using Equation (8A):R _(a) =P ₁·(C1_(av))+P ₂·(C2_(av))+P ₃·(C3_(av))  8Awhere C1av, C2av, and C3av are the average roughness of classes C1 toC3, respectively; and P1, P2, and P3 are the probabilities of matchingto each of classes C1 to C3, respectively, as defined above.

Alternatively, the concrete surface roughness index (AR) may beestimated from the model using Eq. 8B.AR=P ₁(AR1_(a))+P ₂(AR2_(av))+P ₃(AR3_(av))  8Bwhere AR1av, AR2av, and AR3av are the average concrete surface roughnessindex of classes C1 to C3 respectively.

The input image size is set to 300×300 pixels. In this example, the Adamoptimization method (Kingma, et al., (2014), Adam: A method forstochastic optimization, arXiv preprint arXiv:1412.6980; which is herebyincorporated herein by reference in its entirety) with learning rate0.0001 was used to update the weights of the network iteratively. Due tothe small size of the dataset, in this example the batch size wasselected as eight (e.g., eight images are passed through the network ineach iteration to update the network parameters), and the epoch was setto 100. In other words, in each epoch, the entire training dataset ispassed through the network. The loss function was selected as“categorical cross-entropy,” which is a common loss function formulticlass classification.

Example 3: Correlation of Image Processing and Machine Learning to 3DLaser Scanning

The degree of surface roughness of the nine small specimens used in theprevious examples was calculated based on Equation (1), as listed inTable 1, using the 3D laser scanning method. The average roughness,standard deviation, and coefficient of variance were calculated as1.5867 mm (0.0625 in.), 0.3563 mm (0.0140 in.), and 8%, respectively.The nine samples are categorized into three classes C1 (Ra>1.85 mm), C2(1.85 mm≥Ra≥1.35 mm), and C3 (Ra<1.35 mm), representing the roughestsurface to the smoothest. The average surface roughness of each class islisted in Table 1.

TABLE 1 Calculated Ra for each small sample (S). S Class R_(a) (mm)R_(a) (in.) Avg. (C_(av)) 1 C1 2.1999 0.0866 2.1499 mm 2 2.0998 0.08270.0846 in. 3 C2 1.6998 0.0669 4 1.5999 0.0630 1.5499 mm 5 1.4999 0.05910.0610 in. 6 1.3998 0.0551 7 C3 1.3005 0.0512 1.268 mm 8 1.2979 0.05110.0498 in. 9 1.1989 0.0472

The results of the digital image processing method for the images whichwere taken in a dark room with flash applied (FIG. 6) are presentedherein. The total area of exposed aggregate was used as a criterion todistinguish the degree of concrete surface roughness. Not surprisingly,the samples with more exposed aggregates (FIGS. 6A and 6B, samples (S) 1and 2) get higher average surface roughness if compared to samples withless exposed aggregates (FIGS. 6G, 6H, and 6I for samples (S) 7, 8, and9, respectively), as shown in Table 2, which shows the productivity ofthis method. The maximum, minimum, and average of the threshold used foreach image and the total area of aggregate for each image are listed inTable 2.

TABLE 2 Digital image processing results for small samples (S).Aggregate T T T area S (min) (max) (avg.) mm²(in²) (AR) 1 110 160 13710472 (16.23) 0.4509 2  85 170 144  9970 (15.45) 0.4293 3 125 200 151 8946 (13.87) 0.3852 4 140 205 169  7506 (11.63) 0.3232 5 145 200 167 7449 (11.55) 0.3207 6 160 180 161  7431 (11.52) 0.3199 7 160 220 187 4890 (7.58) 0.2105 8 130 220 190  4888 (7.58) 0.2105 9 150 230 196 4874 (7.56) 0.2099 T = threshold.

Considering the test results graphically shown in FIG. 7, thecorrelation between the surface roughness obtained from the 3D laserscanning method (R_(a)) and the ratio of the aggregate area-to-totalarea (AR) can be expressed by regression using a polynomial function(Equation 9) for AR values ranging from 0.2 to 0.45. The correspondingconstant coefficients of the suggested equation are listed in Table 3.R _(a) =k ₁ AR ² +k ₂ AR+k ₃  (9)where Ra is concrete surface roughness (from 3D laser scanning methodfor calibration); AR is the ratio of aggregate area-to-total surfacearea; k1, k2, and k3 are coefficients to correlate the ratio of AR((aggregate area) to (total area)) to R_(a) (concrete surfaceroughness.)

TABLE 3 Coefficients values for k₁, k₂ and k₃. Unit System k₁ k₂ k₃ R²(mm) 15.106 −6.0754 1.8778 0.9668 (in.)  0.5947 −0.2393 0.0739

Results discussed herein are related to images which were taken in adark room with flash applied. For any other environmental or imagingconditions, a new correlation between AR and concrete surface roughnessRa may be established as the method may be sensitive to environmental orimaging conditions.

The ResNet50 deep learning model was trained on the training dataset fortwo models. The first model, with augmentation (aug.), was trained witha total number of 150 images (63 real images and 87 artificiallygenerated, or augmented, images) and the second model, withoutaugmentation (non-aug.), was trained using a total number of 63 realimages, and then the performance of the network was evaluated on thetesting data. The testing samples are not augmented, and they are notused during the training step. Therefore, the model has never seen thetesting data previously. The testing images (two per each surface; 18 intotal) are carefully selected to be different from the training samples(e.g., they are taken in a different lighting condition, or with adifferent angle).

FIG. 8 shows the accuracy performance of these two models (aug. andnon-aug.) on the validation (testing) dataset during training. In otherwords, after each epoch, the model is evaluated on the testing set.

The performance of “non-aug.” model did not improve as much as the “aug”model over the time and the dotted plot for “non-aug.” is relativelyflat (it only improves from 25% to 38%), while the “aug.” model didimprove its accuracy much more over the time and reaches to 94% in thefinal epochs. Similarly, in FIG. 9, it can be noted that the loss valueof the “aug.” model reduces over time; however, the “non-aug.” does notlearn as much from the dataset. Both FIG. 8 and FIG. 9, respectively,show the trends of accuracy and loss for these two models to show howeach model performed after each iteration of optimization.

These results demonstrate the importance of augmentation or syntheticdata in training deep neural networks on such small datasets. Theconfusion matrices of these two models (“aug.” and “non-aug.”) are alsoshown in Tables 4 and 5, respectively. Detailed presentations of Tables4 and 5 are shown in Tables 8 and 9, respectively. From Table 5, it canbe observed that the model without augmentation is strongly over-fittedto the class C3 and cannot detect other classes well.

From Table 4, it can be observed that the model with augmentation (the“aug.” model) can classify all images in the current data set correctlyexcept one of the samples in class 1. On the other hand, “non-aug.”model misclassified many samples from class 1 and class 2 to class 3instead.

TABLE 4 Confusion matrix for deep learning model with transfer learningand data augmentation. Predicted class aug. C1 C2 C3 Real class C1 3 1 0C2 0 8 0 C3 0 0 6

TABLE 5 Confusion matrix for deep learning model with transfer learningand without augmentation. Predicted class non-aug. C1 C2 C3 Real classC1 1 0 3 C2 0 2 6 C3 0 0 6

Example 4: Method Validation Using Large Scale Specimen

To validate both digital image processing and machine learning methods,a new set of images was extracted from a large scale T-beam, which wasroughened using sandblasting, as illustrated (not to scale) in FIG. 10.The purpose of this step is to run new images that both methods neverexperienced before and to predict the degree of surface roughness andthen compare the results to those obtained from the 3D laser scanningmethod.

Six images were used in the validation of both digital image processingand machine learning methods, which belong to a web of a large T-beamwith a total length (L) of 4,724 mm (186 in.) and a height (H) of 419 mm(16.5 in.) A shown in FIG. 10. Each sample (X by Y) was 203.2 mm×203.2mm (8 in.×8 in.), as shown in FIG. 10 and in FIGS. 11A through 11F.

Table 6 shows the values for average surface roughness obtained from the3D laser scanning method (Equation 1), the ratio of the aggregatearea-to-total area AR (Equation 7), and average surface roughness (Ra)obtained from digital image processing method using Equation (9). Themaximum error of the digital image processing method never exceeded 7%.

TABLE 6 Comparison between 3D laser scanning and image processingresults. R_(a) R_(a) scanning AR (mm) Sample (mm) Eq. 7 Eq. 9 Error 11.4141 0.285 1.3727 2.93% 2 1.6791 0.386 1.7827 6.17% 3 1.3591 0.1951.2671 6.77% 4 1.3857 0.266 1.3301 4.02% 5 1.4798 0.332 1.5251 3.06% 61.5577 0.356 1.6287 4.56% (1 mm = 0.03937 in.)

Table 7 shows the probability of each image to match each class (e.g.,large scale sample 2 match 17% of class 1, 66% of class 2, and 17% ofclass 3), the predicted class which is the class with dominatedprobability (e.g., large scale sample 2 matches 66% of class 2,therefore, it was classified as C2, whereas large scale sample 3 matches97% of class 3, therefore, it was classified as C3), and the averagesurface roughness obtained from machine learning using Equation (8). Itcan be noted that the maximum error of the machine learning method neverexceeded 6.5%.

TABLE 7 Comparison between 3D laser scanning and machine learningresults. (1 mm = 0.03937 in.) R_(a) Eq. 8 Sample P1 P2 P3 C (mm) Error 10.01 0.48 0.51 C2 1.4110 0.22% 2 0.17 0.66 0.17 C2 1.6162 3.75% 3 0.000.03 0.97 C3 1.2743 6.24% 4 0.01 0.17 0.82 C3 1.3356 3.62% 5 0.01 0.620.37 C2 1.4507 1.97% 6 0.04 0.90 0.06 C2 1.5568 0.06%

FIG. 12 shows a comparison between the values of average surfaceroughness for both image processing and machine learning methods againstthose obtained from 3D laser scanning with errors never exceeded 7% forboth methods.

FIG. 15 and FIG. 16 show the detailed data supporting Table 4 and Table5, respectively.

These examples demonstrate:

1. The digital image processing method can be used as an efficient toolfor measuring surface roughness if the environmental conditions of theimages are accounted for.

2. The ratio of the aggregate area-to-the total area of the surface (AR)can be used as an index to classify surface roughness and can bedirectly correlated to the average surface roughness obtained from the3D laser scanning method with an accuracy of at least 93%.

3. The machine learning technique may provide advantages includingreduced sensitivity to environmental conditions, and decreased need forhuman intervention.

4. The transfer learning technique could be successfully used totransfer the knowledge learned from a large scale dataset, such asImageNet, to the proposed domain with a small number of images.

5. By applying data augmentation techniques, the proposed deep learningmodel could increase the accuracy by 56% if compared to the modelwithout augmentation (94% vs. 38%) and could overcome issues such as thelimited number of source data for deep learning training.

6. The pretrained ResNet50 augmented model could successfully classifythe surface roughness and could predict the surface roughness with anaccuracy of more than 93% for new images that were not experiencedduring the training of the machine learning model.

When the term “about” is used herein, in conjunction with a numericalvalue, it is understood that the value can be in a range of 95% of thevalue to 105% of the value, i.e. the value can be +/−5% of the statedvalue. For example, “about 1 kg” means from 0.95 kg to 1.05 kg.

It should be understood that the examples and embodiments describedherein are for illustrative purposes only and that various modificationsor changes in light thereof will be suggested to persons skilled in theart and are to be included within the spirit and purview of thisapplication.

All patents, patent applications, provisional applications, andpublications referred to or cited herein are incorporated by referencein their entirety, including all figures and tables, to the extent theyare not inconsistent with the explicit teachings of this specification.

What is claimed is:
 1. A method for obtaining a measurement of surfaceroughness of a concrete sample, the method comprising: receiving, by aprocessor, a first training image set; augmenting, by the processor, thefirst training image set, to create an augmented training image setcomprising training images; determining, by the processor, a surfaceroughness measurement for each training image; classifying, by theprocessor, each training image into a training classification groupbased on the surface roughness measurement determined for thatrespective training image; determining, by the processor, an averagesurface roughness measurement for each training classification group;training, by the processor, a convolutional neural network to classifynew images into a roughness classification group, to produce a trainedconvolutional neural network (TCNN); receiving, by the processor, a newimage of the concrete sample, the new image not being present in theaugmented training image set; and determining, by the processor, usingthe TCNN, a value of average surface roughness (R_(a)) for the new imageof the concrete sample, to obtain the measurement of surface roughnessof the concrete sample.
 2. The method according to claim 1, theaugmenting of the first training image set comprising one or more ofrandom left and right rotation of images, change in brightness, blur,horizontal and vertical flipping, and resizing of images.
 3. The methodaccording to claim 2, the determining of a surface roughness measurementfor each training image comprising at least one of loading values fromexisting measurements or calculating one or more values from digitalimage processing of training images.
 4. The method according to claim 3,the existing measurements comprising non-contact measurements associatedwith respective training images.
 5. The method according to claim 4, thenon-contact measurements comprising 3D laser scanning.
 6. The methodaccording to claim 4, the non-contact measurements comprising digitalimage analysis.
 7. The method according to claim 3, the digital imageprocessing of training images comprising calculation of a ratio ofaggregate area-to-total surface area (AR).
 8. The method according toclaim 7, the training of the convolutional neural network comprisingtransfer learning from a dataset larger than the augmented trainingimage set and training on training images from the augmented trainingimage set.
 9. The method according to claim 1, the determining of theR_(a) for the new image of the concrete sample being completed on acomputing device at a same physical location as the concrete sample. 10.The method according to claim 9, the new image of the concrete samplebeing obtained from a camera in operable communication with thecomputing device at the same physical location as the concrete sample.11. The method according to claim 1, the augmenting of the firsttraining image set being performed in an offline manner.
 12. The methodaccording to claim 1, the determining of the R_(a) for the new image ofthe concrete sample comprising: defining a positive integer n and apositive integer index i ranging from 1 to n; defining a set of nroughness classes (Ci); defining for each Ci an associated averageroughness value (Ci_(av)); generating for each Ci, a probability (P_(i))of matching the new image with that respective Ci, the generating of theP_(i) comprising using the TCNN; and determining the R_(a) for the newimage from the sum of each P_(i) multiplied by the respective Ci_(av).13. The method according to claim 12, the determining of the R_(a) forthe image comprising using the following equation:$R_{a} = {\sum\limits_{i = 1}^{n}{\left( {\left( P_{i} \right) \cdot \left( {Ci_{a\nu}} \right)} \right).}}$14. The method according to claim 12, the augmenting of the firsttraining image set being performed in an offline manner.
 15. The methodaccording to claim 12, the augmenting of the first training image setcomprising one or more of random left and right rotation of images,change in brightness, blur, horizontal and vertical flipping, andresizing of images.
 16. The method according to claim 12, thedetermining of a surface roughness measurement for each training imagecomprising at least one of loading values from existing measurements orcalculating one or more values from digital image processing of trainingimages.
 17. The method according to claim 16, the existing measurementscomprising non-contact measurements associated with respective trainingimages.
 18. The method according to claim 17, the non-contactmeasurements comprising at least one of 3D laser scanning and digitalimage analysis.
 19. The method according to claim 16, the digital imageprocessing of training images comprising calculation of a ratio ofaggregate area-to-total surface area (AR).
 20. A method for obtaining ameasurement of surface roughness of a concrete sample, the methodcomprising: receiving, by a processor, a first training image set;augmenting, by the processor, the first training image set, to create anaugmented training image set comprising training images; determining, bythe processor, a surface roughness measurement for each training image;classifying, by the processor, each training image into a trainingclassification group based on the surface roughness measurementdetermined for that respective training image; determining, by theprocessor, an average surface roughness measurement for each trainingclassification group; training, by the processor, a convolutional neuralnetwork to classify new images into a roughness classification group, toproduce a trained convolutional neural network (TCNN); receiving, by theprocessor, a new image of the concrete sample, the new image not beingpresent in the augmented training image set; and determining, by theprocessor, using the TCNN, a value of average surface roughness (R_(a))for the new image of the concrete sample, to obtain the measurement ofsurface roughness of the concrete sample, the augmenting of the firsttraining image set comprising one or more of random left and rightrotation of images, change in brightness, blur, horizontal and verticalflipping, and resizing of images, the determining of a surface roughnessmeasurement for each training image comprising at least one of loadingvalues from existing measurements or calculating one or more values fromdigital image processing of training images, the existing measurementscomprising non-contact measurements associated with respective trainingimages, the non-contact measurements comprising at least one of 3D laserscanning and digital image analysis, the digital image processing oftraining images comprising calculation of a ratio of aggregatearea-to-total surface area (AR), the training of the convolutionalneural network comprising transfer learning from a dataset larger thanthe augmented training image set and training on training images fromthe augmented training image set, the determining of the R_(a) for thenew image of the concrete sample being completed on a computing deviceat a same physical location as the concrete sample, the new image of theconcrete sample being obtained from a camera in operable communicationwith the computing device at the same physical location as the concretesample, the augmenting of the first training image set being performedin an offline manner, the determining of the R_(a) for the new image ofthe concrete sample comprising: defining a positive integer n and apositive integer index i ranging from 1 to n; defining a set of nroughness classes (Ci); defining for each Ci an associated averageroughness value (Ci_(av)); generating for each Ci, a probability (P_(i))of matching the new image with that respective Ci, the generating of theP_(i) comprising using the TCNN; and determining the R_(a) for the newimage from the sum of each P_(i) multiplied by the respective Ci_(av),and the determining of the R_(a) for the image comprising using thefollowing equation:$R_{a} = {\sum\limits_{i = 1}^{n}{\left( {\left( P_{i} \right) \cdot \left( {Ci_{av}} \right)} \right).}}$